Skip to content

fix(release): run catalogue self-tests without production credentials - #351

Merged
devantler merged 1 commit into
mainfrom
codex/actions-release-tests-348
Oct 1, 2026
Merged

devantler merged 1 commit into
mainfrom
codex/actions-release-tests-348

Conversation

@devantler

Copy link
Copy Markdown
Contributor

🤖 Generated by the Agentic Engineer

Why

Release checks currently receive permission to change live repositories even when only evaluating a possible release. A check failure could therefore affect real repository content.

What

Run the catalogue release checks in disposable repositories while preserving the expected version decisions. Consumer releases and release previews keep their current behavior.

Fixes #348

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
✅ Action performed

Full review finished.

@github-code-quality

github-code-quality Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Code Coverage Overview

Languages: Go, C#

Go / code-coverage/test

The overall line coverage in commit dfcf2c3 in the codex/actions-releas... branch remains at 50%, unchanged from commit 00b5e92 in the main branch.

C# / code-coverage/dotnet

The overall line coverage in commit dfcf2c3 in the codex/actions-releas... branch remains at 100%, unchanged from commit 00b5e92 in the main branch.


Updated October 01, 2026 13:05 UTC

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

✅MegaLinter analysis: Success

Descriptor Linter Files Fixed Errors Max errors Warnings Elapsed time
✅ COPYPASTE jscpd yes no no 0.35s
✅ GO revive 2 0 0 7.16s
✅ REPOSITORY betterleaks yes no no 1.07s
✅ REPOSITORY checkov yes no no 15.23s
✅ REPOSITORY git_diff yes no no 0.03s
✅ REPOSITORY osv-scanner yes no no 0.66s
✅ REPOSITORY secretlint yes no no 0.7s
✅ REPOSITORY syft yes no no 2.23s
✅ REPOSITORY trivy yes no no 10.15s
✅ REPOSITORY trivy-sbom yes no no 0.07s
✅ REPOSITORY trufflehog yes no no 2.59s

See detailed reports in MegaLinter artifacts

Your project could benefit from a custom flavor, which would allow you to run only the linters you need, and thus improve runtime performances. (Skip this info by defining FLAVOR_SUGGESTIONS: false)

  • Documentation: Custom Flavors
  • Command: npx mega-linter-runner@10.1.0 --custom-flavor-setup --custom-flavor-linters COPYPASTE_JSCPD,GO_REVIVE,REPOSITORY_CHECKOV,REPOSITORY_GIT_DIFF,REPOSITORY_BETTERLEAKS,REPOSITORY_OSV_SCANNER,REPOSITORY_SECRETLINT,REPOSITORY_SYFT,REPOSITORY_TRIVY,REPOSITORY_TRIVY_SBOM,REPOSITORY_TRUFFLEHOG

MegaLinter is provided by OX Security
Show us your support by starring ⭐ the repository

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Validation at dfcf2c334851b1ef751dff90d1877128bd1dc9f7:

  • RED: the original workflow failed the new secret-free boundary. GREEN: 28 independent controls reject restored secrets (including bracket/case/serialization syntax), App minting, write grants, production execution, skipped fixtures, and lost required aggregation.
  • All 14 repository baseline commands passed. Existing release configuration/concurrency tests, 37 warning-reader fixtures, ShellCheck/shfmt, README parity, hardening, test wiring, and required-summary guards passed.
  • The actual pinned semantic-release command and shipped configuration selected the expected patch, minor, breaking major, scoped breaking major, and no-release results. Both hook settings passed seven decision cases with local and bare refs unchanged. Four independent command/configuration regressions are rejected; ambient Git paths, templates, and hooks cannot redirect fixture operations.
  • Hosted run 36863973994 has passed both offline release jobs. Their logs show read-only execution and all seven decisions for each hook setting; both production release jobs are skipped. The executed decision processes clear credentials and npm auth configuration. This proves version decisions and flag forwarding, not live notification delivery.
  • Independent final-commit review reports no remaining actionable findings after repairing its two findings. The requested CodeRabbit review and complete hosted CI remain promotion gates.

The normal release command, token scopes, consumer dry-run preview, and production concurrency remain intact. offline-test is an explicit default-off execution mode for catalogue fixtures.

@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Advanced

Run ID: a6bf6ceb-f9d3-4ced-bb71-49c3bb2dadcf

📥 Commits

Reviewing files that changed from the base of the PR and between 00b5e92 and dfcf2c3.

📒 Files selected for processing (6)
  • .github/tests/create-release-fixture.sh
  • .github/tests/test-create-release-fixture-controls.sh
  • .github/tests/test-create-release-offline.sh
  • .github/workflows/ci.yaml
  • .github/workflows/create-release.yaml
  • README.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Recent review details
⏰ Context from checks skipped due to timeout. (113)
  • GitHub Check: [Test] Validate Go Project / 🧹 Lint - mega-linter
  • GitHub Check: [Test] Validate Go Project - Signed Fixes Opted In / 🧹 Lint - mega-linter
  • GitHub Check: zizmor
  • GitHub Check: zizmor
  • GitHub Check: [Test] Enable Auto-Merge - Actor Trust Enabled / eligibility
  • GitHub Check: [Test] Enable Auto-Merge - Actor Trust Enabled / dry-run
  • GitHub Check: [Test] Enable Auto-Merge / dry-run
  • GitHub Check: [Test] Validate Go Project - Default-Branch Maintenance Opted In / 🔍 Detect Changes
  • GitHub Check: [Test] Lint - Passes On Clean Fixture / 🧹 Lint - mega-linter
  • GitHub Check: [Test] Enable Auto-Merge - Pending Queue Enabled / eligibility
  • GitHub Check: [Test] Lint - Manual Workflow Fixes Opted In / 🧹 Lint - mega-linter
  • GitHub Check: [Test] Enable Auto-Merge - Queue Slot 3
  • GitHub Check: [Test] Enable Auto-Merge - Queue Slot 1
  • GitHub Check: [Test] Enable Auto-Merge - Queue Slot 2
  • GitHub Check: [Test] Run .NET Tests - Authenticated Packages / test (windows-latest)
  • GitHub Check: [Test] Run .NET Tests - Authenticated Packages / test (macos-latest)
  • GitHub Check: [Test] Run .NET Tests / test (windows-latest)
  • GitHub Check: [Test] Run .NET Tests / test (macos-latest)
  • GitHub Check: [Test] Create Release - Dry Run / Offline release decisions
  • GitHub Check: [Test] Create Release - No Issue Side Effects / Offline release decisions
  • GitHub Check: [Test] Validate Go Project - Default-Branch Scan Opted In / 🔍 Detect Changes
  • GitHub Check: [Test] Validate Go Project - Default-Branch Test Opted Out / 🔍 Detect Changes
  • GitHub Check: 🧹 Validate manifests
  • GitHub Check: [Test] Validate Go Project - Manual Workflow Fixes Opted In / 🔍 Detect Changes
  • GitHub Check: [Test] Govulncheck Allowlist - Strict Path Blocks (1.27.1, govulncheck-allowlist)
  • GitHub Check: [Test] Govulncheck Allowlist - Strict Path Blocks (1.27.1, govulncheck-generics)
  • GitHub Check: [Test] Govulncheck Allowlist - Strict Path Blocks (1.26.8, govulncheck-generics)
  • GitHub Check: [Test] Govulncheck Allowlist - Strict Path Blocks (1.25.14, govulncheck-allowlist)
  • GitHub Check: [Test] Govulncheck Allowlist - Strict Path Blocks (1.25.14, govulncheck-generics)
  • GitHub Check: [Test] Govulncheck Allowlist - Strict Path Blocks (1.26.8, govulncheck-allowlist)
  • GitHub Check: [Test] Zizmor - Blocks On Vulnerable Fixture
  • GitHub Check: [Test] Enable Auto-Merge - Author Gate
  • GitHub Check: [Test] Enable Auto-Merge - Pentad Gate
  • GitHub Check: [Test] Govulncheck Trigger Guard - Blocks On Bad Input
  • GitHub Check: [Test] Validate Go Project - Test Gate Blocks
  • GitHub Check: [Test] Govulncheck - Timeout & Memory Safety Controls
  • GitHub Check: [Test] Zizmor - Gate Pins Self-Tested Action
  • GitHub Check: [Test] Go Filter Nested Modules
  • GitHub Check: [Test] Govulncheck Allowlist - Allowlisted Advisory Passes (1.25.14, govulncheck-generics)
  • GitHub Check: [Test] Govulncheck Allowlist - Allowlisted Advisory Passes (1.25.14, govulncheck-allowlist)
  • GitHub Check: [Test] Govulncheck Allowlist - Allowlisted Advisory Passes (1.26.8, govulncheck-allowlist)
  • GitHub Check: [Test] Create Release - Least Privilege
  • GitHub Check: [Test] Govulncheck Allowlist - Allowlisted Advisory Passes (1.27.1, govulncheck-generics)
  • GitHub Check: [Test] Validate Go Project - Invocation-Aware Concurrency Key
  • GitHub Check: [Test] Govulncheck Allowlist - Allowlisted Advisory Passes (1.27.1, govulncheck-allowlist)
  • GitHub Check: [Test] Govulncheck Allowlist - Allowlisted Advisory Passes (1.26.8, govulncheck-generics)
  • GitHub Check: [Test] Lint - Gate Pins Self-Tested Image
  • GitHub Check: [Test] Fix Exporter - manual
  • GitHub Check: [Test] Lint - Blocks On Invalid Fixture
  • GitHub Check: [Test] Fix Exporter - export-failure
  • GitHub Check: [Test] Fix Exporter - ordinary
  • GitHub Check: [Test] Fix Exporter - clean
  • GitHub Check: [Test] Validate Go Project - Lint Gate Blocks
  • GitHub Check: [Test] Fix Exporter - read-only
  • GitHub Check: [Test] Go Filter Nested Modules - Blocks On Bad Input
  • GitHub Check: [Test] Govulncheck - Default-Branch & Allowlist Trigger Coverage
  • GitHub Check: [Test] Govulncheck - A Timed-Out Scan Is Retried Once
  • GitHub Check: [Test] Sync Cluster Policies - Allowlist Miss Fails Closed
  • GitHub Check: [Test] Go Suite Trigger Guard - Blocks On Bad Input
  • GitHub Check: [Test] Govulncheck Allowlist - Gate Uses Self-Tested Implementation
  • GitHub Check: [Test] Go Suite - Default-Branch Trigger Coverage
  • GitHub Check: [Test] Delete Workflow Runs - All Workflows / Delete workflow runs
  • GitHub Check: [Test] Delete Workflow Runs - Specific Pattern / Delete workflow runs
  • GitHub Check: [Test] Delete Workflow Runs - Minimal Parameters / Delete workflow runs
  • GitHub Check: test-ensure-gh-skill-script
  • GitHub Check: test-run-dotnet-tests-missing-working-directory
  • GitHub Check: test-update-agent-skills-noop (macos-latest)
  • GitHub Check: test-update-agent-skills-mark-internal (ubuntu-latest)
  • GitHub Check: test-agent-skills-retry-env (macos-latest)
  • GitHub Check: test-update-agent-skills-missing-dir
  • GitHub Check: test-login-to-ghcr-empty-token
  • GitHub Check: test-update-agent-skills-noop (ubuntu-latest)
  • GitHub Check: test-agent-skills-retry-env (ubuntu-latest)
  • GitHub Check: [Lint] Shell Pipeline Assertions
  • GitHub Check: test-update-agent-skills-dry-run
  • GitHub Check: [Test] Run .NET Tests - Coverage Inline Lockstep
  • GitHub Check: test-upload-coverage
  • GitHub Check: [Test] Run .NET Tests - Microsoft.Testing.Platform (macos-latest)
  • GitHub Check: test-update-agent-skills-mark-internal (macos-latest)
  • GitHub Check: [Test] Run .NET Tests - Gate Lockstep
  • GitHub Check: test-retry-script
  • GitHub Check: test-setup-go-toolchain-invalid-version
  • GitHub Check: lint-readme-parity
  • GitHub Check: [Test] Run .NET Tests - Gate Blocks
  • GitHub Check: test-dependency-review
  • GitHub Check: [Test] Upsert Issue - Offline Behavior
  • GitHub Check: test-gh-skill-install-script
  • GitHub Check: test-aggregate-job-checks-empty
  • GitHub Check: lint-ci-coverage-parity
  • GitHub Check: test-setup-go-toolchain (ubuntu-latest)
  • GitHub Check: [Test] Run .NET Tests - Microsoft.Testing.Platform (windows-latest)
  • GitHub Check: test-setup-go-toolchain (macos-latest)
  • GitHub Check: test-setup-agent-skills-pinned
  • GitHub Check: test-setup-agent-skills-missing-input
  • GitHub Check: test-free-disk-space
  • GitHub Check: test-setup-agent-skills-inline (ubuntu-latest)
  • GitHub Check: test-setup-agent-skills-inline (macos-latest)
  • GitHub Check: test-setup-go-toolchain-private-modules
  • GitHub Check: test-run-dotnet-tests (macos-latest)
  • GitHub Check: test-run-dotnet-tests (windows-latest)
  • GitHub Check: test-enable-auto-merge-on-pr
  • GitHub Check: 🧪 Test Guard Installed Skill Edits
  • GitHub Check: test-setup-ksail-cli (ubuntu-latest)
  • GitHub Check: test-diagnose-flux
  • GitHub Check: [Test] Run .NET Tests - Preserve Source Mapping
  • GitHub Check: test-run-dotnet-tests (ubuntu-latest)
  • GitHub Check: test-setup-agent-skills-multi-agent
  • GitHub Check: [Test] Enable Auto-Merge on PR - Credential-Free CI
  • GitHub Check: [Test] Create Issues From TODOs - Offline Wrapper
  • GitHub Check: [Test] Validate Shell Pipelines (macos-latest)
  • GitHub Check: [Test] TODO Scanner - Isolated Image
  • GitHub Check: [Test] Retired Repository Links - Enabled (macos-latest)
  • GitHub Check: [Test] Approve PR - Credential-Free CI
🧰 Additional context used
📓 Path-based instructions (5)
Source excerpt: Every `.github/tests/test-*.sh` is a test entrypoint and must have an explicit invocation in a `ci.yaml` step's `run:` block.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • .github/workflows/ci.yaml
  • .github/tests/test-create-release-fixture-controls.sh
  • .github/tests/test-create-release-offline.sh
Source excerpt: Run [zizmor](https://github.com/zizmorcore/zizmor) to scan for GitHub Actions vulnerabilities

📄 CodeRabbit inference engine (actions/CONTRIBUTING.md)

Files:

  • .github/workflows/create-release.yaml
  • .github/workflows/ci.yaml
Source excerpt: Pin `` to a commit SHA.

📄 CodeRabbit inference engine (README.md)

Files:

  • README.md
Source excerpt: Add a `test-` job to `.github/workflows/ci.yaml` (`persist-credentials: false` on checkout), wired into `ci-required-checks` (both `needs:` and the inline summary step's `JOB_RESULTS` value)

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • .github/workflows/ci.yaml
Source excerpt: Add a row to the Actions table in the catalogue part of [README.md](README.md)

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • README.md
🔇 Additional comments (6)
.github/tests/create-release-fixture.sh (1)

1-88: LGTM!

.github/workflows/create-release.yaml (1)

20-27: LGTM!

Also applies to: 33-33, 135-180

.github/workflows/ci.yaml (1)

3644-3660: LGTM!

Also applies to: 3688-3701

README.md (1)

183-184: LGTM!

Also applies to: 202-206

.github/tests/test-create-release-fixture-controls.sh (1)

1-42: LGTM!

.github/tests/test-create-release-offline.sh (1)

1-106: LGTM!


📝 Walkthrough

Walkthrough

The release workflow adds an offline test mode that runs release-decision cases without production credentials. The fixture uses disposable local Git repositories and checks expected release outcomes and unchanged refs. CI callers use the new mode, and regression scripts check fixture isolation and reject altered workflow controls.

Priority: ➖ Normal

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to dfcf2

The catalogue checks use a separate credential-free, dry-run path while preserving normal consumer release behavior. No concrete merge-blocking risk was identified; merge after normal checks pass.

Security Architecture Review

Security architecture risk: 🔵 Low · up to dfcf2

Release self-tests lose production write credentials and run against disposable local repositories. The reviewed changes show no introduced security concern, but dependency execution and interrupted-run recovery remain incompletely verified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — For the changed CI callers, routine fixture execution loses the ability to mutate live repositories through an App token. Expected Git mutations are confined to per-invocation temporary repositories and local bare remotes. Executed code still has runner-level access; environment stripping and file-only Git transport are not a general process sandbox.

Security Findings and Attack Paths

  • inferred — The reviewed offline path does not expose a supported new route from caller-controlled flags to production release authority: typed inputs select mutually exclusive jobs, dry-run is mandatory, and the fixture accepts only a fixed command template. This assessment does not establish the safety of dependency internals or replace the unknown broader security coverage.

Trust Boundaries and Controls

  • observed — The release subprocess receives a stripped environment, empty npm configuration files, disabled ambient Git configuration, and an explicit file remote. Setup and assertions additionally disable Git hooks and templates. The catalogue release configuration contains only the commit-analyzer plugin, so these tests do not exercise live notification delivery.

Resilience and Maintainability Implications

  • inferred — Unique temporary workspaces and an EXIT cleanup trap separate repository state across ordinary invocations. Ref invariants are checked only after successful release execution; cancellation cleanup and partial-failure state are unverified. The npm cache is reused, although these workflows define no cache restoration or production-job sharing. These limitations do not by themselves establish exposure of production repository authority.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 3 files. (3 skipped: 3… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The PR implements the coding requirements in #348. Both CI catalogue callers use offline-test: true, dry-run: true, contents: read, and no APP_PRIVATE_KEY. The offline workflow job rejects non…
Out of Scope Changes check ✅ Passed The changed files remain within #348. The fixture, boundary-control tests, workflow changes, required-check aggregation, and README changes all implement or verify secret-free catalogue release testin…
Title check ✅ Passed The title clearly and concisely describes the main change: running catalogue release self-tests without production credentials.
Description check ✅ Passed The description explains the security concern and the proposed offline testing approach. It also states that consumer releases and previews retain their current behavior.
Full details: Docstring Coverage

Explanation

Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 3 files. (3 skipped: 3 unsupported.)

  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Assessed the Docstring Coverage warning in CodeRabbit summary 5931907015 at dfcf2c334851b1ef751dff90d1877128bd1dc9f7.

Its full detail says: "Analyzed 3 functions across 3 files. (3 skipped: 3 unsupported.)" All three touched functions are Bash test helpers. Bash has no native function docstring construct; the scripts describe their purpose and the isolation helper has a purpose comment. The report identifies no missing API documentation or source location. Refuted as an unsupported-language metric, with no code change.

body_findings=0-resolved@dfcf2c334851b1ef751dff90d1877128bd1dc9f7

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Promotion readiness revalidated at dfcf2c334851b1ef751dff90d1877128bd1dc9f7, against freshly fetched main 00b5e9235f5dc33ac914e39acaab9cc2511cfee5.

  • Full hosted CI 36863973994 and Code Quality 36863970319 completed successfully at this head. All 270 reported gates are success/skipped.
  • The authenticated CodeRabbit summary reports no actionable comments at this exact head; the canonical verdict is GREEN. The unsupported Bash docstring warning has a specific refutation. Complete thread census: unresolved=0, total=0; no inline or review-body findings remain.
  • Both hosted release variants executed all seven real semantic-release decisions without production credentials, under read-only permissions; production release jobs were skipped. Expected versions and unchanged fixture refs were observed. Local regression and baseline evidence is recorded above.
  • Signed head verified by GitHub, no conflict, and native merge state CLEAN. The required-gate helper reports every readable required check/workflow PASS; its code-quality rule remains UNVERIFIED because that rule exposes no head-analysis API. This is not treated as COMPLETE or stale. The successful hosted analysis and native enforced merge rules remain authoritative.

Promoting now. The ready-for-review CI run will settle before the final exact-head merge check.

@devantler
devantler marked this pull request as ready for review October 1, 2026 13:04
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Final merge check at dfcf2c334851b1ef751dff90d1877128bd1dc9f7: ready-for-review CI 36866081747 completed successfully. All 509 reported checks/statuses across the original and final runs are success/skipped. Fresh main remains 00b5e9235f5dc33ac914e39acaab9cc2511cfee5; head unchanged, trusted author, signed commit, MERGEABLE/CLEAN, unresolved=0 total=0, and current-head CodeRabbit GREEN.

The unchanged unsupported Bash docstring warning remains specifically refuted above; no new finding has appeared. Every readable required gate passes. The code-quality helper still reports UNVERIFIED, not COMPLETE; the successful hosted analysis and GitHub native merge enforcement remain authoritative.

Both hosted offline modes and the local decision/regression tests provide the recorded behavior evidence. Merging with the exact-head guard, then verifying main CI, the normal production release path, and the tagged artifact publication.

@devantler
devantler merged commit bc9fd0b into main Oct 1, 2026
509 checks passed
@devantler
devantler deleted the codex/actions-release-tests-348 branch October 1, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run catalogue release tests without production write credentials

1 participant